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ABSTRACT 



The main goal of Fiddle, a distributed debugging engine, is to provide a flexible platform for developing 
debugging tools. Fiddle provides a layered set of interfaces with a minimal set of debugging functionalities, 
for the inspection and control of distributed and multi-threaded applications. 
I This paper illustrates how Fiddle is used to support integrated testing and debugging. The approach 

Q^ ■ described is based on a tool, called Deipa, that interprets sequences of commands read from an input file, 

generated by an independent testing tool. Deipa acts as a Fiddle client, in order to enforce specific execu- 
tion paths in a distributed PVM program. Other Fiddle clients may be used along with Deipa for the fine 
debugging at process level. Fiddle and Deipa functionalities and architectures are described, and a working 
I example shows a step-by-step application of these tools. 

o ' 

' KEYWORDS: Distributed Debugging, Software Testing, Tool Integration 

o 

> 

X : 1 Introduction 



Developing parallel and distributed applications is still a difficult task, even after about several 
decades of intense research on methodologies and support tools. On one hand, this is due to the 
intrinsic complexity of distributed computing, involving many dynamic interacting entities, execut- 
ing on multiple processors, leading to a global nondeterministic behavior. On the other hand, much 
research and development in this area has been facing a need to continuously adapt to new hard- 
ware, operating system and middleware platforms, and new programming language approaches 
(ranging from imperative to functional or logical, and to object- and agent-oriented models). Such 
constant evolution in the computing models and platforms made many interesting tools obsolete, as 
they were not able to adapt, or their design was too dependent upon specific models or platforms. 

As parallel machines were becoming available since the 80s, parallel debuggers have been devel- 
oped and some of them evolved into significant commercial tools, such as Total View iTooil . Initially, 
such debuggers usually followed a monolithic approach, where a debugging engine and its user 
interface were combined into a single large program. 



In M. Ronsse, K. De Bosschere (eds), proceedings of the Fifth hiternational Workshop on Automated Debugging (AADE- 
BUG 2003), September 2003, Ghent. COmputer Research Repository (http://www.acm.org/corr/), cs.SE/yymmnnn; whole 
proceedings: cs.SE/0309027. 
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Developments on distributed computing have motivated debugging architectures based on the 
client/ server model, such as p2d2 liHoo96J . which clearly separates the user interface from the de- 
bugging engine. Most designs are based on having a separate process in each machine node, to 
support the remote access to the debugging functionalities, and rely upon a central server to do all 
the processing. This was an important design decision to enable the debugging tool to adapt to dif- 
ferent user requirements and environments but the importance of such a step was not recognized 
until recently. Related and significant efforts were also done concerning monitoring of parallel and 
distributed programs ICLV+98,.CD98..LWSB97...LWO96..KP00.TSLW00.KGV96.l . 

An important requirement for flexible development tools is their ability to adapt to new com- 
puting platforms or user requirements. This motivates an approach based on a clear separation be- 
tween the software layers which support the minimal functionalities required by a tool, and the 
layers which provide the extensions that may be required by specific user and application scenar- 
ios. This requires a debugging support platform that enables the inclusion of new tools, to provide 
complementary fimctionalities for application development and acting as intermediaries between 
the debugger and the other tools. 

In previous work, some of the above requirements, involving the integr ation of a testing tool for 
parallel programs (STEPS) and a parallel debugger (DDBG) were tested lLCK~'"97l . However, the 
limitations of such experiment have motivated the continuation of this work towards a more flexible 
debugging architecture. Fiddle. 

In this paper, we first describe the main characteristics of the Fiddle architecture. Next, in Sec.|31 
we discuss an approach for integrated testing and debugging and, in Sec.|4]we illustrate the approach 
through a working example. Then, in Sec.|5l we describe how Deipa was implemented as a Fiddle 
client tool, and we present conclusions and future work in Sec.|6l 

2 The Fiddle Distributed Debugging Engine 

Fiddle ILCOOIILCOll is a distributed debugging engine based on a client-server operational model. 
The debugging engine acts as a server and the debugging tools, which are not included in Fiddle 
specification, act as the client(s). 

The Fiddle debugging engine has the following main characteristics: 

• Support for interactive correctness debugging. Fiddle includes all the traditional debugging facil- 
ities available in sequential debuggers, such as DBX ILin90l and GDB fStaSS'l, but extended 
to operate upon the multiple processes of a distributed program. Some of the services pro- 
vided by Fiddle operate at process level, e.g., breakpointrng and single-stepping, while some 
other operate at application level, e.g., obtaining control of newly created application processes 
{spaivn'ed in a PVM program). As a whole. Fiddle provides a consistent and complete set of ser- 
vices which would be unavailable if a simple set of independent sequential debuggers would 
have been used instead; 

• Client/server model. Multiple debugging interfaces (client tools) can connect simultaneously to 
Fiddle and have access to the same set of target processes, providing, in this way, multiple 
"debugging views" over the same target application; 

• Extensibility. Fiddle functionality may easily be extended by adding new clients, which provide 
specific and/or complementary functionalities to the debugging system; 

• Support of high-level user-defined abstractions. Besides the ability to debug distributed programs 
at textual (source) level. Fiddle can also be applied to programs developed using higher level 
programing languages and / or models, such as visual parallel programming languages, which 
allow a parallel application to be specified in terms of a set of graphical entities describing 
application components and their interconnections. Fiddle extensions can be incorporated into 
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the debugging system, which maps the functionahties required by such high-level debugging 
interface onto Fiddle basic debugging services; 

• Tool synchronization. By allowing multiple concurrent client tools to access a common set of 
target processes. Fiddle needs to provide some basic support for tool synchronization. Using 
Fiddle this can be achieved in two ways: i) as all requests made by client tools are controlled by 
a central daemon. Fiddle is able to serialize the service requests, avoiding some of the interfer- 
ences among these tools; ii) client tools can receive notification events reporting changes in the 
target processes and /or service requests issued by other clients, allowing a tool to be aware of 
the "debugging environment", and to react upon changes^. 

• Easy integration in Parallel Software Engineering Environments. Fiddle extensibility can be ex- 
plored to support the cooperation and integration of debugging and other related tools in the 
environment, e.g., on-line program visualization tools. 



2.1 Fiddle Software Architecture 

Fiddle is structured as a hierarchy of five functional layers, each providing a specific set of debugging 
services and accessible through an interface library. Any layer may be used directly by a client tool 
if its set of services is found to be adequate for the client needs. Each layer is also used indirectly 
by the layer immediately above. For example, in Fig.E Layer 2m has two clients: the tool CT^" and 
Layer 3™. 



Client Tools 






r 








TPi 




v 
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r 
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\ 
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V 




) 



Fiddle 



Target Application 



Figure 1: Fiddle's Layered Architecture 

A service requested by, e.g., a Layer 3m client tool, will be processed and transferred successively 
to the underlying layers, until Layer Os is reached, and the service requested applied to the target 
process. The reply to such service request is also successively processed and transferred to the upper 
layers until the client tool gets the result. 

There is a minimum set of fimctionaUties common to all Fiddle layers, namely: 

• Inspect/control multi-threaded target processes. Fiddle provides a basic set of debugging services to 
act upon threads within a multi-threaded process; 

• Inspect/control mtdtiple target processes concurrently. Any client tool may use Fiddle services to 
inspect and control multiple processes concurrently; 



^This coordination meclianism is not implemented in Fiddle yet. See Sec. 12.21 
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Support for client tool(s). All layers accept, at least, requests from one client tool or from the layer 
above. The upper layers also accept multiple client tools operating concurrently upon the same 
target application. 



Besides these common functionalities, each layer provides a set of specific functionalities. These 
functionalities are supported by the software architecture shown in Fig.|2 



Layer 2„ 




Figure 2: The Fiddle software architecture 



Fiddle layers numbers range from zero to three, with incremental ftmctionality, and have a suffix s 
or m, representing the class of accepted clients: single- or multi-threaded, respectively. 



Layer Os This layer implements a set of local debugging services, including the common set of 
debugging services described above, which are made available to a single client tool. 

The following components are known to this layer: i) Target processes: a subset of processes that 
compose the target application and are being executed in the local node; ii) Node debuggers: associ- 
ated to each target process there is a node debugger, e.g., a GDB instance, which is responsible for 
acting upon it; iii) LayerOg library: provides single-threaded access to the debugging functionalities; 
iv) Client tool: a single-threaded tool accessing Fiddle and possibly implementing a user interface. 



Layer 0,„ This layer extends Layer 0^ to provide support for a single multi- threaded client tool. The 
client may now use threads to concurrently control multiple target processes and the interaction with 
the user. 



Layer 1,„ Both Layers Os/m have access to processes on the local node. Layer 1,„ extends Layers Qg/m 
to allow a client tool to have access to remote target processes. 

The components known to this layer are: i) Target processes: running in the local or remote nodes; 
ii) Node debuggers: one associated to each target process; iii) LOm Server: an instance of this daemon 
will be automatically launched in each node hosting target processes; iv) Layer \„i Library: provides 
thread-safe access to local and remote debugging functionalities; v) Client tool: client tools using this 
layer may transparently access local and remote target processes by using their global identifiers. 
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Layer 2,„ This layer extends Layer 1,„ to provide support for multiple simultaneous client tools, 
which may issue concurrent requests to Fiddle. This is achieved by using a Ll,„ Server daemon, as 
an intermediary between the client tools and the LOm Server daemons. This Ll„i Server daemon mul- 
tiplexes the service requests from the multiple client tools, submits them to Layer l„i, and demulti- 
plexes the corresponding replies back to the clients. 

Layer 3m Layers 0/1/2 services use a blocking semantics, keeping the calling thread waiting for 
the reply. This layer extends Layer 2,„ to provide an event-based interaction between Fiddle and its 
clients. In contrast to the lower layers, a thread that invokes a method in the LayerSm Library doesn't 
block waiting for its reply. Instead, it receives a Request Identifier which will be used later to request 
and process the reply. When the service is executed, its success status and data (the reply) is sent 
to the client as an event. These events may be processed by the client tool in two different ways: 
i) Asynchronous mode: the general methodology to process Fiddle events is to define a handler. When 
a service is executed by Fiddle, a new thread will be created to execute the handler. A structure 
describing the results of the requested service is passed as an argument to the handling function, 
together with the Request Identifier; ii) Synchronous mode; in this mode. Fiddle keeps the notification of 
the event pending until the client tool explicitly requests it by invoking a Fiddle primitive. 

2.2 Fiddle Current Status 

Currently, Fiddle Layers 0/1 /2^y„j are fully implemented and fimctional. Implementation of Layer 3™, 
which will provide event notification and event handling mechanisms for client tools, is ongoing 
work. Fiddle development has taken part on Linux machines, but there are no strong dependencies 
on this operating system. Also, as the Fiddle debugging engine has no user interface, there are no 
dependencies on graphical environments/packages either. 

If the target application is using PVM to spawn (launch) new processes. Fiddle can also automat- 
ically acquire such newly created processes, launching them under the control of a node debugger, 
and making them possible targets for future services requests. 

Along with the Fiddle debugging engine, a set of text oriented debugging interfaces (Fiddle con- 
soles) are being developed, one for each Fiddle layer, which allow to explore the full set of ser- 
vices provided by each layer. Additionally, other Fiddle client tools have been developed, such as 
FG I | Aug021 (the Fiddle Graphical Interface) based in the Gnome Desktop Environment, PADI ISNCOOl 
(a group oriented parallel debugger) based in the Java AWT and Deipa (described in this paper). 

Being still in a prototype stage. Fiddle is currently not publicly available (yet). However, it is 
available on a demand basis to anyone who request it to the author^. 

3 Integrating Testing and Debugging: the DEIPA Approach 

Program correctness requires that a given specification of the intended behavior be satisfied. Al- 
though the development of high-level abstractions for distributed programming has contributed to 
ease the task of program development, there are still many opportunities for programming errors, 
posing the need for support tools. 

Many programming errors can be detected, in a more or less automatic way, through a static anal- 
ysis of the program source text. Such analysis can also assist the programmer in the prediction of the 
program behavior concerning specific correctness properties. However, the program source text is 
not always available. Also, the global program behavior is often the result of a combination of be- 
haviors, depending on the operating system or runtime support layers of a computing environment. 

This explains the significance of approaches for dynamic analysis, which are centered upon the 
observation of real execution. Such approaches assume, from the beginning, the incomplete nature 

•^E-mail: jml@di.fct.unl.pt 
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of this process, due to the usually huge number of computation states which can be generated by 
distributed program execution. As such, they are typically based on the selection of a finite set of 
representative test configurations, followed by an observation of the results of program execution. 
The definition of such test scenarios is, of course, dependent upon the classes of errors or program 
properties that one is trying to check. 

The main goal of a debugging tool is to help analysing erroneous program behavior, possibly 
identified by a previous testing stage, and to assist the programmer in the formulation or confirma- 
tion of hypotheses on the causes of errors, and in the tracing of their origins in the program text. 

Testing and debugging are naturally intertwined. On one hand, testing helps identifying errors 
whose causes must later be traced in a debugging stage. On the other hand, after a successful de- 
bugging session, one typically needs to reconsider the set of test configurations. Debugging can also 
help identifying unforeseen situations which may require the design of new testing scenarios. 

The above aspects have been recognized for a long time, leading to many proposals of method- 
ologies and tools for combined testing and debugging. See LKW991ICKW00I for a more complete 
survey on this topic . 

Deipa lLCK+971 was initially developed to support a testing-and-debugging development cycle, 
by allowing the composition of two separately developed tools, STEPS l„KW96J and DDBG ICLA99I 
ICLD98I . The current configuration of such testing and debugging architecture is illustrated in Fig. |3] 




Aplication 
File 

Environment 
Target Process 



Figure 3: Tool composition of STEPS and Fiddle using Deipa 

Based on a static and djmamic analysis of the target program, STEPS generates a behavior spec- 
ification file, the TeSS file. This file includes the definition of a list of global breakpoints (a consistent 
collection of local breakpoints). Some of those breakpoints have special instructions to modify one or 
more variable values. This is necessary to drive correctly the application execution when in presence 
of conditional branches or loops, e.g., on a i f statement. 

To force the application execution to conform to the specification in the TeSS file, Deipa generates 
sequences of debugging (control and inspection) commands to Fiddle, setting a local breakpoint in 
each target process and individually driving them until the breakpoints are reached. Once a process 
is stopped in a breakpoint, its internal status (e.g., variable contents) may be changed, according to 
the TeSS specification. 

At any time, the Fiddle ability to handle multiple clients may be explored, and the user may switch 
to another tool, such as Fiddle's debugging console or graphical interface (FGI 1 Aug02|) and perform 
a more detailed inspection and control of the target processes. This allows the user to perform de- 
bugging at distinct abstraction levels, e.g., at a global program level to inspect process interactions, 
and at local process level, to inspect individual process execution. 

On completing such a fine (process-level) debugging, the user may return to Deipa and proceed 
with the controlled execution mode, or release /stop the target application if no more debugging is 
needed. 
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4 A Working Example 

To demonstrate how Deipa works, we have built a simple application. It is composed of two pro- 
grams: the echo_client, which sends a number to the server and waits for its reply; and the 
echo_server, which gets a number from the client and exits if the received number is even, or 
sends -1 back otherwise. 

Since it is common for distributed applications to have a program that initializes the environment, 
we consider that the echo_client is the startup program, launching the echo_server server via 
pvm_spawn ( ) . 

First, we shall present the source code of both PVM programs, with echo_client on the left 
column and echo_server on the right one. The tiny leftmost number indicates the line nimiber, 
used for future reference. 

echo_client 

1 #include <pvm3.h> 

2 #include <stdio.h> 

3 #include <stdlib.h> 

4 #include <unistd.h> 

5 
6 
7 

8 int main () 
' I 

10 int mytid ; 

11 int totid=0; 

12 int value ; 

13 
14 

15 mytid = pvm_mytid ( ) ; 

16 

17 value = pvm_spawn ( " echo_server " , NULL, 

18 PvmTaskDebug , " . " , 1 , & t o t i d ) ; 

19 

20 /* Sending will force the server to 

21 exit , and the client will wait 
11 forever for the reply */ 

23 

24 value = 0; 

25 pvm_initsend (0); 

26 pvm_pkint ( & mytid ,1 , 1 ) ; 

27 pvm_pkint (& value , 1 , 1 ) ; 

28 pvm_send ( totid , 1); 

29 

30 /* Get the reply from server */ 

31 pvm_recv (—1,-1); 

32 pvm_upkint (& value, 1 ,1); 

33 printf ( " Received^value^%d\n" , value); 

34 

35 pvm_exit ( ) ; 

36 

37 return (0); 

38 } 

The most relevant source lines in the client code are line 17, where the server is launched (spawned); 
line 28, where the message is sent to the server; and line 31, where the client blocks waiting for the 
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echo_server 



1 #include <pvm3.h> 

2 #include <stdio.h> 

3 #include <stdlib.h> 

i 

5 

6 int main () 

7 I 

8 int mytid ; 

9 int from, value; 

10 

11 mytid = pvm_mytid ( ) ; 

12 

13 pvm_recv ( — 1 , — 1 ) ; 

14 pvm_upkint (&from,l ,1); 

15 pvm_upkint (& value , 1 , 1 ); 

16 

17 if (( value%2)==0) 

18 exit ( ) ; 

19 else 

20 value = — 1; 

21 

22 pvm_initsend (0); 

23 pvm_pkint (&value , 1 , 1); 

24 pvm_send ( from , 1 ) ; 

25 

26 pvm_exit ( ) ; 

27 

28 return 0; 

29 } 
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reply. 

On the server, the most relevant source lines are line 13, where the server waits for a message 
from the client; line 17, where the server decides how to react to the received message (terminate or 
reply); and line 24, where the server replies to the client (if this was the case). 

As the development of the STEPS tools was discontinued by its authors, and Deipa uses TeSS 
files as one of its inputs, we produce the following TeSS files ourselves, which describe a testing 
scenario for the echo client and server processes. The tiny line numbers are not part of the TeSS file 
itself but were included only for future reference to specific lines. 



1 START_FILE: 

2 echo_client 

3 

4 SPAWN_TABLE: 



1 echo_client echo_client.c 13142 4, 

1 16 1 2 echo server echo server . c 59634 2 



INITIAL : 



(1 ,1 ,17) )] , 
(1 ,1 ,28) 1] , 
(2 ,1 ,28) 1] , 
(2 ,1 ,28) ,(1,2,13) 1] , 
(2 ,1 ,28) ,(2,2,13) 1] , 
(2,1 ,28) ,(1 ,2 ,17 ,[2 ,2 , 
(2 ,1 ,28) ,(2,2,17) 1] , 
(1 ,1 ,31) ,(1 ,2 ,24) 1] , 
(2 ,1,31) ,(2,2 ,24) j] , 
(1 ,1 ,35) ,(1,2 ,26) j] , 
(2 ,1,35) ,(2,2 ,26) 1]; 



'value" ,"1" ]) )] 



Each line represents a global breakpoint, as a collection of local breakpoints of the form " ( T , I , L ) ", 
which are the type of breakpoint, the virtual identifier of the process (VID) and the line number 
respectively. The type of breakpoint indicates if the program should stop before (code 1) or after 
(code 2) the selected line number. The virtual identifier of the target process will be mapped by 
Deipa to PVM task IDs at runtime. The new process, at line 14 of the TeSS file, is captured by the 
launcher program and integrated into the Fiddle and Deipa execution environment, as described in 
Sec.|Hl 

The TeSS file specifies global breakpoints in the following points: 

• Pvm_spawn ( ) , to ensure that new processes are, indeed, created on the PVM environment; 

• Send and receive messages, to enable the verification of emission and reception of messages; 

• Pvm_exit () , to ensure that all processes get out correctly. 

4.1 Using Deipa and Fiddle to capture errors 

The main idea of this example is to launch the distributed application and control its execution from 
the Deipa console; and to examine individual processes state on the Fiddle text user interface (f 2m 
console). We must notice that this example requires the usage of Layer 2,„, as there are multiple client 
tools simultaneously connected to Fiddle, namely Deipa, the launcher and a Fiddle console. 
The screen shots in Figures HI |5| and |6] refer respectively to the following situations: 



Fifth Int. Workshop on Automated and Algorithmic Debugging 



CONTROL AND DEBUGGING OF DISTRIBUTED PROGRAMS USING FIDDLE 



151 



• The capture of the echo_server target process, created at line 17 of echo_client, and its 
integration into Deipa and Fiddle; 

• Driving the application behavior by forcing its execution path; and 

• The reception of the answer at the client side. 

4.1.1 The capture of the new process 

This task consists of intercepting the spawn action, the launching of the new target processes under 
the control of a node debugger, and the notification of Deipa and Fiddle on the existence of the new 
target process. 

FigureHlshows three windows, the PVM console on the top left, the Fiddle Layer 2„i console below, 
and Deipa on the right. On each window two states are shown. On the first state, the echo_server 
process hasn't been launched yet; while on the second, when the new process was already started 
(although not running). 



V vr[n'gJ|ocalliasr/1in|i/aad6bug/filial_v&rsiQH/SQurce_CQde/WQ[kilig_i — n X W vrTnglocalhQ5l:/tmp/aadebijgrfinal_version/5Qurcg_CDde^TOrkinq. 



[wrroQloc^lhost wor-king_exaMple]$ pwrii 



HOST TID FLdG <ll COIIimNIl 

loc^lhost, loealdcimain 40002 4/c - 



P4^> ps 



HOST TID FLdG <ll COIIimNIl 

localhost. loealdcimain 40002 4/c - 



loc^lhost.lotaldoKiain 40005 
pvra^ U 



2/f echio_5&r-yer 



V vrni^locdlha5l:/tnip/aadtbiig/final_ver5iaii/sourc:e_code^working_- 



[urrtiplocalhost iyorkin9_ei:a[rpLe]t f2«i 



I FIDDLE - vO.5.1 



(Level 2m console) 



I Copyright (c) 1399 DI - FCT/UML 
I 

I Type 'help' on prawpt to get help 

I <TrtE> esp^rds corrirri^nds 

I fibbneui.3ticrs are accepted if unambiguous 



f2ir []> tids 

TID PTT 



fSn []> tids 
TID 



TP_PID 






LLB_PID 
3109 



LLD_PID 
3109 
3139 



L_TII HBCHINE 

1 localhast. lonaldomsin 



L_TI1 NfiCHINE 

1 localhast^ lonaldomsin 

2 lo^allipft. localdpi^ain 



f2» []> D 



[urnSlocalhost uorkirig.eKaniple]! ^./console_delpa echo_example,tes 



(Lei^el 2rr ( 



:ole) 



Cgpyright (t) 2W2 Dl - FCT/UML 

Type 'help' or prompt ta get help 

<TflE> espands coromands 

flbbreyiatior? ^re accepted if unawbigijous 



2 proGC 
] ^ 



^ one process 

con5ole_deipa > step 

pth_nian^9er ; **« begin of process threads' list 

( 1) preu=[bptype= 1 line= 17] ^ctu5l=[bptype= 1 line= 28] - 
pth_man59er; *** ftf^d af process thread's list *** 

pth_laLjrcheri connection from 127.040^.32634, fdi/.^ 

pth_laurcher: /honie/vrm/pwmBr'bin/LIhlUWr'echo.server 
pth_laurcher: [tid=2. '^id=2] \ 

new connection 

con£ole_deipa > step 

pth_niana9er : *** begin of process threads' list 
(■ 1) preu=[bptype= 1 line= 28] actual=[bptype= 2 lins= 28] 
I :.i preu^Ebptype^rJULL line=HULL] actual=[bptype=HlLL line^rJJLL] 

pth_roan^9er; **« end of process thread's list **« 



console.deipa > |] 



two 



Figure 4: The capture of a new process 



Two of the commands available on the Deipa console are: run, to start execution the distributed 
application, and step, to advance the distributed application to the next global breakpoint indicated 
in the TeSS file. 

On Deipa console, the distributed application (only echo_client for now) starts running and 
stops before line 17, the pvin_spawn ( ) function call. When we step into the next global breakpoint, 
the echo_client process is spawned. As the step command lists all the known processes before 
doing any action, its possible to verify that at the first step there is just one process (corresponding 
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to the echo_client program), while at the second step there are aheady two processes, meaning 
the echo_server was aheady laimched and is now imder control of Fiddle (and Deipa). 

Since Layer 2,„ level allows several concurrent clients to connect to Fiddle, we also laimched a 
Fiddle console. Once again, in the initial state (before the pvm_spawn ( ) ), there is only one process 
(with TID 1), but in the second state there are two processes (with TIDs 1 and 2), corresponding to 
echo_client and echo_server, respectively. 

Similarly, in the top left window (the PVM console), one can notice that in the initial state there 
is a single PVM task, and after the second step there are two PVM tasks. 

From the observation of the target application with these three different tools, we may confirm 
that the new process was launched correctly and integrated into the debugging environment of Fid- 
dle and Deipa. 



4.1.2 Driving the application behavior 

As stated before, the server program will exit if it receives an even number, leaving the client blocked 
forever waiting for a non-existing reply. We can change this behavior and force the server to always 
reply to the client, no mater it receives an odd or and even, by forcing the contents of the value 
variable at line 17 of echo_server to always be an odd number. 

The Fig. |5] shows the same three windows as before. The PVM console om the to left, the Fiddle 
Layer 2„i console just below, and Deipa on the right. 



V vrini^localhost:^tmp/addebug/final_version^^ijrce_code/working_i _ □ X 



V vrni*iJ|QCilho5C/lni|j/aa(lcl)ug/firal_versior/50urct_code/workiiig_ 



pvm> ps 

HOST TU FLUE Ol CaHMUND 

pvm> ps 

HOST TU FLUE Ol CaHMUND 

localhost, localdoMain 40002 4/c - 

localhast, localdoiiain 40003 echa.seruer 
pvm!> W 



V vrmi^ local ho 



i:g;finaLversion/sourte_code/woirking_i 



[yroEloc^lhost Ljork.in9_example]5 f2m 



FIIIiLE - 



CLeyel 2rii console) 



Copyright [c] 1999 DI - FCT^Ur^L 

Type 'help' on prorrpt to get help 

<Tf1E> expands coirrfiands 

rtbbreviatioi^s ar-e accepted if unarobiguous 



[]> tids 

TII ATT TP.PID LLE_PIIi L.TID HACHIME 

In 2736 1 localhost^locsldoriisin 

[]y tids 

TII ATT TP.PID LLE_PIIi L_TID HACHIME 

In 2736 1 localhost^locsldoriisin 

£ n 27G6 2 lcicalho5t,.localdoriiain 

[]> 2 evaluate value 

$1 = 13451G282 



[]!> 2 evaluate value 

i2 = 1 



- old value (not initialized] 
■ new value 



[urrrtQlocalhcst i.]orkiri3_e^a[rtpIe]t ,/coh^sole_ deipa echo_&^anpIe|.tes 



^Leuel 2m console) I 



I Copyright ic] 2m DI - FCTr^UHL 



Type 'help' or pr-arript to get help 

<lf\E> e:<p5rd5 coriiitands 

Abbreviations are accepted if Linambiguous 



con5ole_deip5 > run 

console_deipa > step 

pth_iii^r^ger: *** begin of process threads' list *** 

( 1) ph-eu^[bpt!jpe- 1 line- 17] actual-[bpt!jpe- 1 liris- 29] 
pth.iiiarager: *** end of process thread's list 
pth.Iauncher: connection froro 127.0.0.1.32826. fd:7 
pth_ launcher; /hane/urrr/purHS/bin/LIHLjf/echo.ssrvsr 
pth_Uuncher; [tid=2^ uid=3] 

ccnscle_deipa > step 

pth_iii^r^ger: *** begin of process threads' list *** 
C V pr-eu=[bpt!jpe= 1 line= 28] actual=[bpt!jpe= 2 line= 28] 
C 2) pre4^[bpt!dpe^NULL line^MULL] actual^[bpt!dpe^NULL line^MULL] 
pth.inarager: *** end of process thread's list *** 

console_deipd > step 

pth.irt^ragert *** begin of process thre^ids' list *** 
. 1) prew=[bptype= 2 Iine= 28] actLiaI=[bptype= 2 line= 28] 
C 2) prev-[bptype-NULL line-MULL] actual-Ebptype- i line- 13] 

pth_iii^r^ger: *** end of process thread's list *** 

console_deipa > step 

pth_iii^rager: begin of process threads' list 

C 1) preu=[bptype= 2 line= 28] sctu^l=[bptype= 2 line= 28] 

C 2) pr'eu=[bpt!jpe= 1 line= 13] actual=[bpt!jpe= 2 line= 15] 

pth.iiiar^geri *** end of process thread's list *** 

console_deip^ > step 

pth.iii^rager: *** begin of process threads' list *** 
C 1) pr-eu-[bpt!jpe- 2 lirie- 28] actual-Ebptype- 2 lirie- 28] 
C 2) prev-ibptype- 2 line- 13] actual-Ebptype- 1 line- 17] 
pth_iii^rager: end of process thread's list m*m 
set_dl l_uarg_fLjnc!: setuar ualue=l 



con3ole_deip^ > W 



setting a new vaiue tor the 
"vaiue' variable 



Figure 5: Driving the application behavior 



Deipa was used to control the application until global breakpoint at line 15 is reached. At this 
point, we may use Fiddle console to inspect the value variable, which is reported to have the garbage 
value of 134516282. 
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Lets step to the next (line 16 of TeSS file) global breakpoint using Deipa, and re-inspect the value 
variable. Although this variable has been initialized to zero at line 15 of echo_server, it should 
contain now the value 1. This is achieved by the special global breakpoint of TeSS line 16, which also 
defines the new contents of the value variable. The contents of the value variable are checked again 
on the Fiddle console, and the value 1 confirmed. 



4.1.3 Receiving the answer at the client side 

In order to verify that the client always receives a reply message, we need to execute line 31 of the 
client's code. If the server has exited, the client will remain blocked forever at this line. 

As shown in Fig. |6l the Fiddle console on the left window informs that the client is stopped at 
line 32, and Deipa on the right window shows that the client stopped after executing line 31 (it is 
type 2 global breakpoint). 



V vim(^lacalFiast:flm|i/aadebug/final_VBrsion/wun:&_code/warking_4 _ □ X 



V vrmgi|QcalhQsl:/lmp/aadcb»g/niial_v6rsiQn/SQurc6_CQii6/working_t 



[yrroGloc^ilhost wcirking_e>;=i^ple]$ pyra 
pwra> p3 

HOST TID PLUG <li CONHdMIl 

loc^lhost.lotaldoKisin 40002 4/c - 

pwni> ps 

HOST TID PLUG <li CONHdMIl 

Iccalhost^ lotaldarrtain 40002 4/c - 

localhost. lotaldcimain 40003 £/f echo_serwer 

pwni> 



vrni^locAlhQsl:/tmp/aadebLig/fiiiaLver5iQn/SQurc€^Qde/WQrkinfl_< . 



□ X 



FIDDLE - vD.5.1 CLewel 2ir console) 

Copyrisht (c) 1399 DI - FCT/UHL 

Type 'help' en pr-owpt to get help 

<TflD> e;;p5rd5 cerfirfianids 

Abbrewi^tiors are accepted if unambiguous 



P2w []> tids 

TID UTT 

1 n 

P2w []> tids 

TID fllT 

1 n 



TP.PID 





TP.PID 





LLLPID L_TI1 HflCHINE 

273G 1 localhcst. localdomain 



LLD_PID L_TI1 HflCHINE 

273G 1 localhast. locaLdomain 

276B 2 localhast. locsidom^in 



- cli ent received th e m essage 



t-2[^ []> 2 ^v^l^^^te valus 
Jl = 134516282 

f2m []> 2 eu.3luate ^^alue 

— ■ J2 - 1 

f2iii []> 1 irfe-line y'^^ 

Line 32 of "echo_client,c" starts at address 0x8049441 <maln+233> 
and ends at CIKSC49454 <riiain+252>, 
^^/tpp/^adebiig/f irial version/souree codeyijonkinig e^;ar^pLe/echo client, c;32;557;be< 

G043441 

KB []> D 



console_deipa > step 

pth_roan^3er ; **« begin of process threads' hit 

( i) preu=[bptLjpe= 1 line= 17] actual=[bptype= 1 lirie= 2S] 
pth_nian59er : **« end of process thread's list **« 
pth_laurii;her; connection from 127, 0,0.,!, 32826, fd;7 
pth_laijrclieri /hopie/vr-n/pvnS/bin/LINUJ^/echc.server 
pt.h_laLjrcheri [tid-2, VLd-2] 

console_deipa > step 

pth_riian59er ; **« b&gin of process threads' list «** 
( i) preu-[bptype- 1 line- 28] actual-Lbptype- 2 line- 28] 
( 2) preu=[bptype=rJULL line=NULL] actual=[bptype=HLLL line=HJLL] 

pth_niana9er ; **« end of process thread's list **« 

con3ole_deipa > step 

pth.nian^ger ; VVV begin of process threads' list 
C 1) preu-[bptype- 2 line- 28] actual-[bptype- 2 line- 28] 
( 2) preu=[bptLjpe=nULL lin&=HULL] actual=[bptype= 1 lin&= 15] 

pth_nan^3«r; **« end vi' process thread's list **« 

□nsole_deips > step 

pth_nian5ger I #** begin of process threads' list *** 

1> prev=[bptype= 2 line= 23] ^ctu^l=[bptype= 2 line= 23] 

2) pre\y-[bptype- 1 line- 13] sctusl-[bptype- 2 line- 13] 

pth_nian^9er ; #** end of process thread's list #** 

con£ole_deipa > ^tep 

pth_man59er: *** begin of process threads' list *** 
\ 1) preu-[bptype- 2 line- 25] actual-[bptype- 2 line- 28] 
2) preu=[bptype= 2 line= IS] ^ctu5l=[bptype= 1 line= 17] 

pth_man59er; *** ftf^d of" process thread's list *** 

5et_all_v5rs_fLincJ seti/ar value-1 

:onsole_deipa > step 

pth_riian59er ; **« begin of process threads' list «** 

. 1) prev-[bptype- 2 line- 28] actual-[bptype- 2 line- 2S] 

2^ preu=[bptLjpe= 1 line= 17] actual=[bptype= 2 line= 17] 

pth_nian59er : **« end of process thread's list **« 

console_deipa > step 

pth_nian53er ; *** begin of process threads' list 

1) preu=[bptype= 2 line= 28] actual=[bptype= 1 line= 31] 

2) preu=[bptype= 2 line= 17] actual=[bptype= 1 line= 24] 

pth_riian59er ; **« end of process thread's list **« 



client receives the message 



con5ole_deipa > step 

pth_niana9er ; **« begin of process threads' list «** 

1> pre^J=[bptype= 1 line= 51] actual=[bptype= 2 line= 51] 

2) prev-[bptype- 1 line- 24] sctusl-Ebptype- 2 line- 24] 

pth_nian^9er ; end of" process thread's list 

server sends the message 



J 



consoIe_deipa > [\ 



Figure 6: Receiving the answer at the client side 

The above example shows how a set of separately developed tools: Deipa and Fiddle (and also 
the PVM console) can be used to drive a distributed application according to a specification of its 
desired behavior, which may differ from the application behavior in an uncontrolled environment. 



5 Deipa as a Fiddle Client 

Above Layer 2,„, Fiddle's API supplies functions to manage multiple clients on a single Fiddle ses- 
sion, interacting concurrently with Fiddle (and with the target application). Deipa, Fiddle's Layer 2^ 
console and FGI are examples of such clients. 
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5.1 The DEIPA tool 

The Deipa tool is composed of two programs: i) the main program, which interacts with the user 
and controls the application behavior; and ii) the launcher, which captures newly created application 
processes (currently supporting only PVM) and makes their registration on Deipa and Fiddle. 

The main program From Fiddle perspective, Deipa is just another client tool accessing the target 
application. To become a Fiddle client, and able to access its services, Deipa must be linked to the 
Fiddle Layer 2,„ library and call the registration function available in the library API. 

The launcher To capture new PVM processes and integrate them in the debugging environment 
(Deipa and Fiddle), the PvmTaskDebug flag must be activated when the target application spawns 
a new process, and the PVM_DEBUGGER environment variable must specify the launcher location. In 
this way, instead of spawning the new process directly, PVM spawns the launcher, which receives the 
name of the process to la unch as a command line argument (for more details on this topic, consult 
the PVM documentation IGBD+941 ). 

Once the launcher is running, it executes the following steps: 

1. Creates a bi-directional communication channel, for the future interactions with Fiddle; 

2. Announces itself to Fiddle as a node debugger, indicating that future interactions should use 
the communication channel; 

3. Annoimces itself to Deipa, informing on the new target process name and Fiddle identification 
number; 

4. Redirects its standard I/O channels to the above mentioned communication channel; 

5. Core image auto-replacement (exec) with a node debugger, running the new target process. 
The redirected I/O channels are inherited on exec'ing, and the node debugger will have its 
I/O redirected to the communication channel. 

The Fiddle API includes a function that allows to register /incorporate an existing node debugger 
into the Fiddle environment. The Deipa launcher fits into this category, but before becoming a node 
debugger, it supplies some relevant information to Deipa. 

5.2 Inside Deipa 

Deipa is structured in two main parts: i) the front-end, currently only a text-oriented user interface 
(Deipa console) is implemented; and ii) the hack-end, where all the relevant Deipa functionalities are 
implemented. 

The Deipa console The commands available to the user can be divided in two sets: i) the ones 
whose scope is limited to Deipa itself, e.g., open (load a new TeSS file), and ii) those that act upon 
the target processes, e.g., run and step (drive the target application imtil the first/ next global break- 
point). 

The Deipa library Deipa library, whose architecture is described in Fig. |7| uses multiple threads, 
one for each target process (the process threads) and three internal management threads. 

The process threads are responsible for the control and inspection of target processes. 

The three management threads are: i) the main thread, responsible for receiving requests, verifying 
if a request is valid in the current state of the library, and sending the replies; ii) the manager thread, 
responsible for managing aU the process threads; and iii) the launcher thread, responsible for receiving 
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DEIPA Library 




DEIPA Launcher 

Figure 7: Deipa implementation 



the registration of each new process created by the pvm_spawn ( ) and signaled by the launcher 
program. 

Details on Deipa internals can be found in IMor02l . 



6 Conclusions and Future Work 

The Fiddle engine meets several requirements for the debugging of multi-thread and multi-process 
distributed applications. Its suitability has been evaluated through the development of prototypes 
to test the integration of several types of support tools. This includes the support for graphic debug- 
ging interfaces, the integration of testing and debugging, and the integration of visualization and 
debugging tools. 

In this paper, we have described how Fiddle eases the development of integrated testing and 
debugging support. This is achieved through a separate tool, Deipa, which acts as an intermediary 
between an independent testing tool (STEPS) and the Fiddle debugging engine. Furthermore, Fiddle 
provides support for concurrent clients, allowing the user to perform debugging and control of a dis- 
tributed application at two distinct levels: the Deipa level, where global program states are followed, 
and at the Fiddle level, where local process states can be inspected and modified. 

Ongoing work on Fiddle client tools, such as FGI and PAD!, will lead to new experiments on 
tool integration and cooperation, with the increased user friendliness given by the graphical user 
interfaces. There are plans to do a similar work on Deipa, replacing its text console with a much 
friendlier graphical interface. 

Communication between the client tools and Fiddle, and internally between Fiddle components, 
has been reformulated and changed from a proprietary data format to XML. This usage of the XML 
standard will improve Fiddle ability to integrate with third party software packages. 
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